Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimization package #343

Merged
merged 32 commits into from
Aug 1, 2022
Merged

Optimization package #343

merged 32 commits into from
Aug 1, 2022

Conversation

uprendis
Copy link
Contributor

@uprendis uprendis commented Jul 25, 2022 •
edited
Loading

  • add multidb framework, which allows to configure DBs and tables layout based on routing rules. It allows to tune performance by splitting/merging DBs, configuring DB type and DB cache amount
  • add batching for genesis processing, which speeds up genesis blocks processing by ~2x
  • DBs files descriptors limits is configurable and scaled by default according to OS process limit (fix File Descriptor limit #323)
  • add caching for event IDs
  • add caching for LLR voting weights
  • during fullsync, LLR syncing is active only if events are already synced up
  • fix nil cachescale issue after loading toml config file

@uprendis uprendis requested review from hadv, cyberbono3 and rus-alex July 25, 2022 18:22
@uprendis uprendis requested a review from andrecronje as a code owner July 25, 2022 18:22
This was referenced Jul 25, 2022
Merged
Closed
hadv
hadv reviewed Jul 27, 2022
View reviewed changes
cmd/opera/launcher/launcher.go Outdated Show resolved Hide resolved
cmd/opera/launcher/check.go Outdated Show resolved Hide resolved
cmd/opera/launcher/dbcmd.go
log.Info("Stats before compaction", "db", name)
showLeveldbStats(db)
log.Info("Stats before compaction", "db", humanName)
showLeveldbStats(db)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we might rename this method to showDBStats() for generic name in multi db

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, but after we merge your changes for new statter. For now, this method indeed works only for leveldb

@hadv
Copy link
Contributor

hadv commented Jul 28, 2022 •
edited
Loading

@uprendis I rebased the PR with develop branch and cannot start the opera node (from scratch)

panic: runtime error: invalid memory address or nil pointer dereference
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x4a44bb8]

goroutine 1 [running]:
github.com/Fantom-foundation/go-opera/gossip.(*handler).Stop(0xc00085e600)
        /Users/hadv/fantom/go-opera/gossip/handler.go:725 +0x1b8
github.com/Fantom-foundation/go-opera/gossip.(*Service).Stop(0xc0001fb000)
        /Users/hadv/fantom/go-opera/gossip/service.go:485 +0x139
github.com/ethereum/go-ethereum/node.(*Node).stopServices(0xc00013b450, {0xc0000c0fd0, 0x1, 0xc0006a2fc8?})
        /Users/hadv/go/pkg/mod/github.com/!fantom-foundation/[email protected]/node/node.go:293 +0xbe
github.com/ethereum/go-ethereum/node.(*Node).Close(0xc00013b450)
        /Users/hadv/go/pkg/mod/github.com/!fantom-foundation/[email protected]/node/node.go:213 +0x18e
github.com/Fantom-foundation/go-opera/cmd/opera/launcher.makeNode.func3()
        /Users/hadv/fantom/go-opera/cmd/opera/launcher/launcher.go:367 +0x38
panic({0x50fffa0, 0x6037d40})
        /usr/local/Cellar/go/1.18.3/libexec/src/runtime/panic.go:838 +0x207
github.com/Fantom-foundation/go-opera/gossip.(*Service).RecoverEVM(0xc0001fb000)
        /Users/hadv/fantom/go-opera/gossip/c_block_callbacks.go:493 +0xe4
github.com/Fantom-foundation/go-opera/gossip.(*Service).Start(0xc0001fb000)
        /Users/hadv/fantom/go-opera/gossip/service.go:436 +0x16b
github.com/ethereum/go-ethereum/node.(*Node).Start(0xc00013b450)
        /Users/hadv/go/pkg/mod/github.com/!fantom-foundation/[email protected]/node/node.go:184 +0x3d7
github.com/ethereum/go-ethereum/cmd/utils.StartNode(0xc0000ba420, 0xc00013b450)
        /Users/hadv/go/pkg/mod/github.com/!fantom-foundation/[email protected]/cmd/utils/cmd.go:70 +0x2a
github.com/Fantom-foundation/go-opera/cmd/opera/launcher.startNode(0xc0000ba420?, 0xc00013b450)
        /Users/hadv/fantom/go-opera/cmd/opera/launcher/launcher.go:391 +0x54
github.com/Fantom-foundation/go-opera/cmd/opera/launcher.lachesisMain(0xc0000ba420)
        /Users/hadv/fantom/go-opera/cmd/opera/launcher/launcher.go:274 +0xe5
gopkg.in/urfave/cli%2ev1.HandleAction({0x5097ec0?, 0x53fbd98?}, 0xc000295c00?)
        /Users/hadv/go/pkg/mod/gopkg.in/urfave/[email protected]/app.go:490 +0x5a
gopkg.in/urfave/cli%2ev1.(*App).Run(0xc00021e9c0, {0xc00003c070, 0x7, 0x7})
        /Users/hadv/go/pkg/mod/gopkg.in/urfave/[email protected]/app.go:264 +0x71c
github.com/Fantom-foundation/go-opera/cmd/opera/launcher.Launch(...)
        /Users/hadv/fantom/go-opera/cmd/opera/launcher/launcher.go:252
main.main()
        /Users/hadv/fantom/go-opera/cmd/opera/main.go:11 +0x35

And a test FAILED aslo when running go test ./...

FAIL    github.com/Fantom-foundation/go-opera/cmd/opera/launcher [build failed]

@hadv
Copy link
Contributor

hadv commented Jul 28, 2022 •
edited
Loading

And a test FAILED aslo when running go test ./...

FAIL    github.com/Fantom-foundation/go-opera/cmd/opera/launcher [build failed]

ah, this is just need to update the config_custom_test.go file after rebase code with develop branch

@uprendis
Copy link
Contributor Author

@uprendis I rebased the PR with develop branch and cannot start the opera node (from scratch)

Got you, will rebase and fix it

uprendis added 20 commits July 28, 2022 16:58
@uprendis
add caching for event IDs
4ab95b1
@uprendis
add caching for LLR voting weights
00ddec3
@uprendis
fix keeping mutating memory from it.Value()
aae378a
@uprendis
erase support for outdated DB migrations
d224554
@uprendis
adjust genesis processing ETA logging
7985f36
@uprendis
fullsync syncs LLR only if synced up events
16bc869
@uprendis
stricter synced condition
13f3e01
@uprendis
add support of multidb config
5d4da77
@uprendis
add db migrate command
8fea1c2
@uprendis
adopt fixdity command for multidb feature
0b6e1cd
@uprendis
add batching for genesis processing
c5657e1
@uprendis
use piecefunc from lachesis-base
82d1815
@uprendis
refactor evm.Commit args
d697ba7
@uprendis
add support of 3 DB integrity modes (flushable, flagged, direct)
00186e4
@uprendis
add automatic EVM recover after abrupt stopping
430a0d8
@uprendis
add automatic migration from pre-multi-db layout
b568d12
@uprendis
rename db migrate to db transform
e369a09
@uprendis
rename db fix to db heal
b408c39
@uprendis
tip to use db heal command
60f6524
@uprendis
reduce scaling of MaxNonFlushedSize with RAM size
c192e0c
@uprendis uprendis force-pushed the feature/optimization-pack branch from c3034aa to 53eddab Compare July 28, 2022 09:59
@uprendis
Copy link
Contributor Author

@uprendis I rebased the PR with develop branch and cannot start the opera node (from scratch)

It should be fixed now after 53eddab

hadv
hadv approved these changes Jul 28, 2022
View reviewed changes
Copy link
Contributor

@hadv hadv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be fixed now after 53eddab

Great, I worked for me!

@uprendis uprendis merged commit 11c9b85 into Fantom-foundation:develop Aug 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hadv hadv hadv approved these changes

@cyberbono3 cyberbono3 Awaiting requested review from cyberbono3

@rus-alex rus-alex Awaiting requested review from rus-alex

@andrecronje andrecronje Awaiting requested review from andrecronje andrecronje is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@uprendis @hadv